RESEARCH HIGHLIGHT: Data assimilation in high dimensions

Data assimilation concerns the recovery of a hidden process through partial sequential observations. Classical methods like the Kalman filter can be derived by the Bayes formula. But they are numerically unfeasible when the underlying dimension reaches several million, which is the usual case for weather forecast. The ensemble Kalman filter (EnKF) has been proposed by meteorologists using the idea of Monte Carlo, and finds very good forecast skills with a few hundred samples. But how does EnKF beats the curse of dimensionality remains an intriguing mystery. Tong Xin and his collaborators intent to understand this phenomenon through rigorous analysis. One natural question of EnKF is whether the associated stochastic process is stable. Surprisingly the answer is not always yes, and it does not rely on the sample size. Observable energy dissipation [1] reveals to be a sufficient condition for stability, and violation of it may lead to catastrophic filter divergence [2]. Fortunately, this intrinsic problem can be fixed by a simple adaptive inflation procedure [3]. It is more interesting and difficult to quantify EnKF performance. One approach is to investigate the notion of effective dimension, which is assumed to be low in many practical problems. It is now proved [4] that a low effective dimension can be exploited in EnKF operation, so the the necessary sample size relies only on the effective dimension, but not the state dimension.